{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {},
  "cells": [
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "   <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/master/examples/annotation_import/video.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
        "</td>\n",
        "\n",
        "<td>\n",
        "\n",
        "<a href=\"https://github.com/Labelbox/labelbox-python/tree/master/examples/annotation_import/video.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
        "</td> "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Video Annotation Import\n",
        "\n",
        "* Annotations must be created and uploaded using NDJSON\n",
        "* Supported annotations that can be uploaded through the SDK:\n",
        "    * Bounding box\n",
        "    * Point\n",
        "    * Polyline \n",
        "    * Radio classifications \n",
        "    * Checklist classifications \n",
        "    *  Segmentation masks\n",
        "* **NOT** supported:\n",
        "    * Polygons \n",
        "\n",
        "Please note that this list of unsupported annotations only refers to limitations for importing annotations. For example, when using the Labelbox editor, segmentation masks can be created and edited on video assets."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "### Setup"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "!pip install -q \"labelbox[data]\""
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "import uuid\n",
        "from PIL import Image\n",
        "import requests\n",
        "import base64\n",
        "import labelbox as lb\n",
        "import labelbox.types as lb_types\n",
        "from io import BytesIO\n",
        "import pprint\n",
        "pp = pprint.PrettyPrinter(indent=4)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Replace with your API key \n",
        "Guides on [Create an API key](https://docs.labelbox.com/docs/create-an-api-key)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Add your api key\n",
        "API_KEY=\"\"\n",
        "client = lb.Client(api_key=API_KEY)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Supported annotations for video\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "### Bounding box: (frame-based)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Confidence scores are not supported for frame specific bounding box annotations and VideoObjectAnnotation class\n",
        "\n",
        "# bbox dimensions\n",
        "bbox_dm = {\n",
        "  \"top\":617,\n",
        "  \"left\":1371,\n",
        "  \"height\":419,\n",
        "  \"width\":505\n",
        "}\n",
        "\n",
        "# Python Annotation\n",
        "bbox_annotation = [\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_video\",\n",
        "    keyframe=True,\n",
        "    frame=13,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm[\"left\"], y=bbox_dm[\"top\"]), # x = left, y = top\n",
        "          end=lb_types.Point(x=bbox_dm[\"left\"] + bbox_dm[\"width\"], y=bbox_dm[\"top\"] + bbox_dm[\"height\"]), # x= left + width , y = top + height\n",
        "      )\n",
        "  ),\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_video\",\n",
        "    keyframe=True,\n",
        "    frame=19,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm[\"left\"], y=bbox_dm[\"top\"]),\n",
        "          end=lb_types.Point(x=bbox_dm[\"left\"] + bbox_dm[\"width\"], y=bbox_dm[\"top\"] + bbox_dm[\"height\"]),\n",
        "      )\n",
        "  )\n",
        "]\n",
        "\n",
        "\n",
        "# NDJSON\n",
        "bbox_annotation_ndjson = {\n",
        "    \"name\" : \"bbox_video\",\n",
        "    \"segments\" : [{\n",
        "        \"keyframes\" : [\n",
        "            {\n",
        "              \"frame\": 13,\n",
        "              \"bbox\" : bbox_dm\n",
        "           },\n",
        "           {\n",
        "              \"frame\": 19,\n",
        "              \"bbox\" : bbox_dm\n",
        "           }\n",
        "        ]\n",
        "      }\n",
        "    ]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Point (frame-based)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python Annotation\n",
        "point_annotation = [\n",
        "    lb_types.VideoObjectAnnotation(\n",
        "        name = \"point_video\",\n",
        "        keyframe=True,\n",
        "        frame=17,\n",
        "        value = lb_types.Point(x=660.134, y=407.926),\n",
        "        )\n",
        "]\n",
        "\n",
        "# NDJSON\n",
        "point_annotation_ndjson = {\n",
        "    \"name\": \"point_video\",\n",
        "    \"segments\": [{\n",
        "        \"keyframes\": [{\n",
        "            \"frame\": 17,\n",
        "            \"point\" : {\n",
        "                \"x\": 660.134 ,\n",
        "                \"y\": 407.926\n",
        "            }\n",
        "        }]\n",
        "    }]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Polyline (frame-based)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "######## Polyline ########\n",
        "\n",
        "# Python Annotation\n",
        "polyline_annotation = [\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"line_video_frame\",\n",
        "    keyframe=True,\n",
        "    frame=5,\n",
        "    segment_index=0,\n",
        "    value=lb_types.Line(\n",
        "          points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]\n",
        "      )\n",
        "  ),\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"line_video_frame\",\n",
        "    keyframe=True,\n",
        "    frame=20,\n",
        "    segment_index=0,\n",
        "    value=lb_types.Line(\n",
        "          points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]\n",
        "      )\n",
        "  ),\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"line_video_frame\",\n",
        "    keyframe=True,\n",
        "    frame=24,\n",
        "    segment_index=1,\n",
        "    value=lb_types.Line(\n",
        "          points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]\n",
        "      )\n",
        "  ),\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"line_video_frame\",\n",
        "    keyframe=True,\n",
        "    frame=45,\n",
        "    segment_index=1,\n",
        "    value=lb_types.Line(\n",
        "          points=[lb_types.Point(x=680, y=100), lb_types.Point(x=100, y=190)]\n",
        "      )\n",
        "  )\n",
        "\n",
        "]\n",
        "\n",
        "# NDJSON\n",
        "polyline_frame_annotation_ndjson = {\n",
        "  \"name\": \"line_video_frame\",\n",
        "  \"segments\": [\n",
        "      {\n",
        "        \"keyframes\": [\n",
        "          {\n",
        "            \"frame\": 5,\n",
        "            \"line\": [{\n",
        "              \"x\": 680,\n",
        "              \"y\": 100\n",
        "            },{\n",
        "              \"x\": 100,\n",
        "              \"y\": 190\n",
        "            },{\n",
        "              \"x\": 190,\n",
        "              \"y\": 220\n",
        "            }]\n",
        "          },\n",
        "          {\n",
        "            \"frame\": 20,\n",
        "            \"line\": [{\n",
        "              \"x\": 680,\n",
        "              \"y\": 180\n",
        "            },{\n",
        "              \"x\": 100,\n",
        "              \"y\": 200\n",
        "            },{\n",
        "              \"x\": 200,\n",
        "              \"y\": 260\n",
        "            }]\n",
        "          }\n",
        "        ]\n",
        "      },\n",
        "      {\n",
        "        \"keyframes\": [\n",
        "          {\n",
        "            \"frame\": 24,\n",
        "            \"line\": [{\n",
        "              \"x\": 300,\n",
        "              \"y\": 310\n",
        "            },{\n",
        "              \"x\": 330,\n",
        "              \"y\": 430\n",
        "            }]\n",
        "          },\n",
        "          {\n",
        "            \"frame\": 45,\n",
        "            \"line\": [{\n",
        "              \"x\": 600,\n",
        "              \"y\": 810\n",
        "            },{\n",
        "              \"x\": 900,\n",
        "              \"y\": 930\n",
        "            }]\n",
        "          }\n",
        "        ]\n",
        "      }\n",
        "    ]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Classification: Radio and checklist (frame-based)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python Annotation\n",
        "radio_annotation = [\n",
        "    lb_types.VideoClassificationAnnotation(\n",
        "        name=\"radio_class\",\n",
        "        frame=9,\n",
        "        segment_index=0,\n",
        "        value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = \"first_radio_answer\"))\n",
        "    ),\n",
        "    lb_types.VideoClassificationAnnotation(\n",
        "        name=\"radio_class\",\n",
        "        frame=15,\n",
        "        segment_index=0,\n",
        "        value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = \"first_radio_answer\"))\n",
        "    )\n",
        "]\n",
        "\n",
        "## NDJSON\n",
        "frame_radio_classification_ndjson = {\n",
        "    \"name\": \"radio_class\",\n",
        "    \"answer\": { \"name\": \"first_radio_answer\", \"frames\": [{\"start\": 9, \"end\": 15}]}\n",
        "}\n",
        "\n",
        "#Python annotation\n",
        "checklist_annotation= [\n",
        "    lb_types.VideoClassificationAnnotation(\n",
        "        name=\"checklist_class\",\n",
        "        frame=29,\n",
        "        segment_index=0,\n",
        "        value=lb_types.Checklist(\n",
        "            answer = [\n",
        "                lb_types.ClassificationAnswer(\n",
        "                    name = \"first_checklist_answer\"\n",
        "                ),\n",
        "                lb_types.ClassificationAnswer(\n",
        "                    name = \"second_checklist_answer\"\n",
        "                )\n",
        "            ]\n",
        "            )\n",
        "        ),\n",
        "    lb_types.VideoClassificationAnnotation(\n",
        "        name=\"checklist_class\",\n",
        "        frame=35,\n",
        "        segment_index=0,\n",
        "        value=lb_types.Checklist(\n",
        "            answer = [\n",
        "                lb_types.ClassificationAnswer(\n",
        "                    name = \"first_checklist_answer\"\n",
        "                ),\n",
        "                 lb_types.ClassificationAnswer(\n",
        "                    name = \"second_checklist_answer\"\n",
        "                )\n",
        "            ]\n",
        "            )\n",
        "        )\n",
        "]\n",
        "\n",
        "\n",
        "## NDJSON\n",
        "frame_checklist_classification_ndjson = {\n",
        "    \"name\": \"checklist_class\",\n",
        "    \"answer\": [\n",
        "        { \"name\": \"first_checklist_answer\" , \"frames\": [{\"start\": 29, \"end\": 35 }]},\n",
        "        { \"name\": \"second_checklist_answer\" , \"frames\": [{\"start\": 29, \"end\": 35 }]}\n",
        "  ]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Classification: Checklist and radio (global)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "##### Global Classifications #######\n",
        "\n",
        "# Python Annotation\n",
        "## For global classifications use ClassificationAnnotation\n",
        "global_radio_annotation = [lb_types.ClassificationAnnotation(\n",
        "    name=\"radio_class_global\",\n",
        "    value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = \"first_radio_answer\"))\n",
        ")]\n",
        "\n",
        "# NDJSON\n",
        "global_radio_classification_ndjson = {\n",
        "    \"name\": \"radio_class_global\",\n",
        "    \"answer\": { \"name\": \"first_radio_answer\"}\n",
        "}\n",
        "\n",
        "# Python annotation\n",
        "global_checklist_annotation=[lb_types.ClassificationAnnotation(\n",
        "  name=\"checklist_class_global\",\n",
        "  value=lb_types.Checklist(\n",
        "      answer = [\n",
        "        lb_types.ClassificationAnswer(\n",
        "            name = \"first_checklist_answer\"\n",
        "        ),\n",
        "        lb_types.ClassificationAnswer(\n",
        "            name = \"second_checklist_answer\"\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        " )]\n",
        "\n",
        "# NDJSON\n",
        "global_checklist_classification_ndjson = {\n",
        "    \"name\": \"checklist_class_global\",\n",
        "    \"answer\": [\n",
        "        { \"name\": \"first_checklist_answer\" },\n",
        "        { \"name\": \"second_checklist_answer\"}\n",
        "  ]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Classification: Nested radio and checklist (global)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "########## Nested Global Classification ###########\n",
        "\n",
        "# Python Annotation\n",
        "nested_radio_annotation =[lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_radio_question\",\n",
        "  value=lb_types.Radio(\n",
        "    answer=lb_types.ClassificationAnswer(\n",
        "      name=\"first_radio_answer\",\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_radio_question\",\n",
        "          value=lb_types.Radio(\n",
        "            answer=lb_types.ClassificationAnswer(\n",
        "              name=\"first_sub_radio_answer\"\n",
        "            )\n",
        "          )\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        "  )\n",
        ")]\n",
        "\n",
        "# NDJSON\n",
        "nested_radio_annotation_ndjson = {\n",
        "  \"name\": \"nested_radio_question\",\n",
        "  \"answer\": {\"name\": \"first_radio_answer\",\n",
        "  \"classifications\" : [\n",
        "    {\"name\": \"sub_radio_question\", \"answer\": {\"name\": \"first_sub_radio_answer\"}}\n",
        "   ]\n",
        "  }\n",
        "}\n",
        "\n",
        "# Python Annotation\n",
        "nested_checklist_annotation = [lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_checklist_question\",\n",
        "  value=lb_types.Checklist(\n",
        "    answer=[lb_types.ClassificationAnswer(\n",
        "      name=\"first_checklist_answer\",\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_checklist_question\",\n",
        "          value=lb_types.Checklist(\n",
        "            answer=[lb_types.ClassificationAnswer(\n",
        "            name=\"first_sub_checklist_answer\"\n",
        "          )]\n",
        "        ))\n",
        "      ]\n",
        "    )]\n",
        "  )\n",
        ")]\n",
        "\n",
        "# NDJSON\n",
        "nested_checklist_annotation_ndjson = {\n",
        "  \"name\": \"nested_checklist_question\",\n",
        "  \"answer\": [{\n",
        "      \"name\": \"first_checklist_answer\",\n",
        "      \"classifications\" : [\n",
        "        {\n",
        "          \"name\": \"sub_checklist_question\",\n",
        "          \"answer\": {\"name\": \"first_sub_checklist_answer\"}\n",
        "        }\n",
        "      ]\n",
        "  }]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Classification: Free-form text"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "######### Free text classification ###########\n",
        "text_annotation = [lb_types.ClassificationAnnotation(\n",
        "  name=\"free_text\",  # must match your ontology feature\"s name\n",
        "  value=lb_types.Text(answer=\"sample text\")\n",
        ")]\n",
        "\n",
        "text_annotation_ndjson = {\n",
        "  \"name\": \"free_text\",\n",
        "  \"answer\": \"sample text\",\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Bounding box with sub-classifications (frame-based)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Confidence scores are not supported for frame specific bounding box annotations with sub-classifications\n",
        "\n",
        "# bounding box dimensions\n",
        "bbox_dm2 = {\n",
        "  \"top\": 146.0,\n",
        "  \"left\": 98.0,\n",
        "  \"height\": 382.0,\n",
        "  \"width\": 341.0\n",
        "}\n",
        "\n",
        "# Python Annotation\n",
        "frame_bbox_with_checklist_subclass_annotation = [\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_class\",\n",
        "    keyframe=True,\n",
        "    frame=10,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm2[\"left\"], y=bbox_dm2[\"top\"]), # x = left, y = top\n",
        "          end=lb_types.Point(x=bbox_dm2[\"left\"] + bbox_dm2[\"width\"], y=bbox_dm2[\"top\"] + bbox_dm2[\"height\"]), # x= left + width , y = top + height\n",
        "      )\n",
        "  ),\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_class\",\n",
        "    keyframe=True,\n",
        "    frame=11,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm2[\"left\"], y=bbox_dm2[\"top\"]),\n",
        "          end=lb_types.Point(x=bbox_dm2[\"left\"] + bbox_dm2[\"width\"], y=bbox_dm2[\"top\"] + bbox_dm2[\"height\"]),\n",
        "      ),\n",
        "    classifications=[\n",
        "                lb_types.ClassificationAnnotation(\n",
        "                    name=\"checklist_class\",\n",
        "                    value=lb_types.Checklist(answer=[lb_types.ClassificationAnswer(\n",
        "                        name=\"first_checklist_answer\")])\n",
        "                )\n",
        "            ]\n",
        "  ),\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_class\",\n",
        "    keyframe=True,\n",
        "    frame=13,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm2[\"left\"], y=bbox_dm2[\"top\"]),\n",
        "          end=lb_types.Point(x=bbox_dm2[\"left\"] + bbox_dm2[\"width\"], y=bbox_dm2[\"top\"] + bbox_dm2[\"height\"]),\n",
        "      ),\n",
        "    classifications=[\n",
        "              lb_types.ClassificationAnnotation(\n",
        "                  name=\"checklist_class\",\n",
        "                  value=lb_types.Checklist(answer=[lb_types.ClassificationAnswer(\n",
        "                      name=\"second_checklist_answer\")])\n",
        "              )\n",
        "          ]\n",
        "  )\n",
        "]\n",
        "\n",
        "frame_bbox_with_checklist_subclass_annotation_ndjson = {\n",
        "    \"name\": \"bbox_class\",\n",
        "    \"segments\": [{\n",
        "        \"keyframes\": [\n",
        "            {\n",
        "            \"frame\": 10,\n",
        "            \"bbox\": bbox_dm2\n",
        "          },\n",
        "          {\n",
        "          \"frame\": 11,\n",
        "            \"bbox\": bbox_dm2,\n",
        "            \"classifications\": [\n",
        "              {\n",
        "                \"name\": \"checklist_class\",\n",
        "                \"answer\": [{\"name\": \"first_checklist_answer\"}]\n",
        "              }\n",
        "            ]\n",
        "          },\n",
        "          {\n",
        "          \"frame\": 13,\n",
        "            \"bbox\": bbox_dm2,\n",
        "            \"classifications\": [\n",
        "              {\n",
        "                \"name\": \"checklist_class\",\n",
        "                \"answer\": [{\"name\": \"second_checklist_answer\"}]\n",
        "              }\n",
        "            ]\n",
        "          }\n",
        "        ]\n",
        "      }\n",
        "    ]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Masks (frame-based)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "def extract_rgb_colors_from_url(image_url):\n",
        "    response = requests.get(image_url)\n",
        "    img = Image.open(BytesIO(response.content))\n",
        "\n",
        "    colors = set()\n",
        "    for x in range(img.width):\n",
        "        for y in range(img.height):\n",
        "            pixel = img.getpixel((x, y))\n",
        "            if pixel[:3] != (0,0,0):\n",
        "                colors.add(pixel[:3])  # Get only the RGB values\n",
        "\n",
        "    return colors"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Raster Segmentation (Byte string array)\n",
        "##  For this example we are going to to pass all the annotations payload in a single VideoMaskAnnotation\n",
        "\n",
        "\n",
        "# Single mask\n",
        "url = \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/frame_24_composite_mask.png\"\n",
        "response = requests.get(url)\n",
        "img_bytes = base64.b64encode(response.content).decode('utf-8')\n",
        "\n",
        "# We are generating our frames and instances in this step, and will later add them to the VideoMaskAnnotation that will contain\n",
        "# all frames and instances\n",
        "frames_mask_single=[\n",
        "    lb_types.MaskFrame(\n",
        "        index=20,\n",
        "        im_bytes=response.content # Instead of bytes you could also pass an instance URI : instance_uri=url\n",
        "    )\n",
        "]\n",
        "instances_mask_single=[\n",
        "    lb_types.MaskInstance(color_rgb=(76, 104, 177), name= \"video_mask\")\n",
        "]\n",
        "\n",
        "\n",
        "## Add multiple masks using multiple tools in different frames - Note that only once composite mask can exist per frame\n",
        "frames_cp_mask_url = [\n",
        "    {\"1\": \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/frame_1_composite_mask.png\"},\n",
        "    {\"24\": \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/frame_24_composite_mask.png\"},\n",
        "    {\"26\": \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/frame_26_composite_mask.png\" }\n",
        "]\n",
        "\n",
        "rgb_mask_tool = [(227, 135, 126) ,(169, 248, 152),(83, 152, 103)]\n",
        "cp_masks = []\n",
        "unique_colors = set()\n",
        "\n",
        "\n",
        "lb_frames = []\n",
        "lb_instances = []\n",
        "counter = 0\n",
        "\n",
        "for d in frames_cp_mask_url:\n",
        "    for frame_no, v in d.items():\n",
        "        response = requests.get(v)\n",
        "        colors = extract_rgb_colors_from_url(v)\n",
        "        for color in colors:\n",
        "            if not color in unique_colors:\n",
        "                unique_colors.add(color)\n",
        "                name = \"video_mask\" if color in rgb_mask_tool else \"mask_with_text_subclass\"\n",
        "                lb_instances.append(lb_types.MaskInstance(color_rgb=color, name=name))\n",
        "                counter += 1\n",
        "        lb_frames.append(\n",
        "            lb_types.MaskFrame(\n",
        "                index=frame_no,\n",
        "                im_bytes=response.content\n",
        "            )\n",
        "        )\n",
        "cp_masks.append(lb_types.VideoMaskAnnotation(\n",
        "    frames=lb_frames + frames_mask_single,\n",
        "    instances=lb_instances + instances_mask_single\n",
        "))\n",
        "\n",
        "pp.pprint(lb_frames)\n",
        "pp.pprint(cp_masks)\n",
        "\n",
        "\n",
        "\n",
        "# NDJSON - single tool\n",
        "video_mask_ndjson_bytes_2 = {\n",
        "    'masks': {\n",
        "      'frames': [\n",
        "          {\n",
        "              \"index\" : 31,\n",
        "              \"imBytes\": img_bytes,\n",
        "          },\n",
        "          {\n",
        "              \"index\" : 34,\n",
        "              \"imBytes\": img_bytes,\n",
        "          }\n",
        "      ],\n",
        "      'instances': [\n",
        "          {\n",
        "              \"colorRGB\" : [76, 104, 177],\n",
        "              \"name\" : \"video_mask\"\n",
        "          }\n",
        "      ]\n",
        "    }\n",
        " }"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Multiple instances of bounding box annotations in the same frame"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Fist instance of bounding box ranging from frame 22 to 27\n",
        "bbox_annotation_1 = [\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_video\",\n",
        "    keyframe=True,\n",
        "    frame=22,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm[\"left\"], y=bbox_dm[\"top\"]), # x = left, y = top\n",
        "          end=lb_types.Point(x=bbox_dm[\"left\"] + bbox_dm[\"width\"], y=bbox_dm[\"top\"] + bbox_dm[\"height\"]), # x= left + width , y = top + height\n",
        "      )\n",
        "  ),\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_video\",\n",
        "    keyframe=True,\n",
        "    frame=27,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm[\"left\"], y=bbox_dm[\"top\"]),\n",
        "          end=lb_types.Point(x=bbox_dm[\"left\"] + bbox_dm[\"width\"], y=bbox_dm[\"top\"] + bbox_dm[\"height\"]),\n",
        "      )\n",
        "  )\n",
        "]\n",
        "# NDJSON example:\n",
        "bbox_frame_annotation_ndjson = {\n",
        "    \"name\": \"bbox_video\",\n",
        "    \"segments\": [{\n",
        "        \"keyframes\": [\n",
        "          {\n",
        "            \"frame\": 22,\n",
        "            \"bbox\":  bbox_dm\n",
        "          },\n",
        "          {\n",
        "            \"frame\": 27,\n",
        "            \"bbox\": bbox_dm2\n",
        "          }\n",
        "\n",
        "        ]\n",
        "      }]\n",
        "}\n",
        "\n",
        "# Second instance of bounding box ranging from frame 22 to 27\n",
        "bbox_annotation_2 = [\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_video\",\n",
        "    keyframe=True,\n",
        "    frame=22,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm2[\"left\"], y=bbox_dm2[\"top\"]),\n",
        "          end=lb_types.Point(x=bbox_dm2[\"left\"] + bbox_dm2[\"width\"], y=bbox_dm2[\"top\"] + bbox_dm2[\"height\"]),\n",
        "      )\n",
        "  ),\n",
        "  lb_types.VideoObjectAnnotation(\n",
        "    name = \"bbox_video\",\n",
        "    keyframe=True,\n",
        "    frame=27,\n",
        "    segment_index=0,\n",
        "    value = lb_types.Rectangle(\n",
        "          start=lb_types.Point(x=bbox_dm2[\"left\"], y=bbox_dm2[\"top\"]),\n",
        "          end=lb_types.Point(x=bbox_dm2[\"left\"] + bbox_dm2[\"width\"], y=bbox_dm2[\"top\"] + bbox_dm2[\"height\"]),\n",
        "      )\n",
        "  )\n",
        "]\n",
        "# NDJSON\n",
        "bbox_frame_annotation_ndjson2 = {\n",
        "      \"name\": \"bbox_video\",\n",
        "      \"segments\": [{\n",
        "          \"keyframes\": [\n",
        "            {\n",
        "              \"frame\": 22,\n",
        "              \"bbox\": bbox_dm\n",
        "            },\n",
        "            {\n",
        "              \"frame\": 27,\n",
        "              \"bbox\": bbox_dm2\n",
        "            }\n",
        "          ]\n",
        "        }]\n",
        "    }\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## End-to-end example: Import pre-labels or ground truth"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "### Step 1: Import data rows into Catalog"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "global_key = \"sample-video-jellyfish.mp4\"\n",
        "asset = {\n",
        "    \"row_data\": \"https://storage.googleapis.com/labelbox-datasets/video-sample-data/sample-video-2.mp4\",\n",
        "    \"global_key\": global_key,\n",
        "    \"media_type\": \"VIDEO\"\n",
        " }\n",
        "\n",
        "dataset = client.create_dataset(\n",
        "    name=\"video_demo_dataset\",\n",
        "    iam_integration=None # If this argument is removed, labelbox will use the default integration for your organization.\n",
        ")\n",
        "task = dataset.create_data_rows([asset])\n",
        "task.wait_till_done()\n",
        "print(f\"Failed data rows: {task.failed_data_rows}\")\n",
        "print(f\"Errors: {task.errors}\")\n",
        "\n",
        "if task.errors:\n",
        "    for error in task.errors:\n",
        "        if 'Duplicate global key' in error['message'] and dataset.row_count == 0:\n",
        "            # If the global key already  exists in the workspace the dataset will be created empty, so we can delete it.\n",
        "            print(f\"Deleting empty dataset: {dataset}\")\n",
        "            dataset.delete()"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Step 2: Create/select an ontology\n",
        "Your project should have the correct ontology setup with all the tools and classifications supported for your annotations, and the tool names and classification instructions should match the `name` fields in your annotations to ensure the correct feature schemas are matched.\n",
        "\n",
        "For example, when we create the bounding box annotation above, we provided the `name` as `bbox_video`. Now, when we setup our ontology, we must ensure that the name of my bounding box tool is also `bbox_video`. The same alignment must hold true for the other tools and classifications we create in our ontology.\n",
        "\n",
        "\n",
        "[Documentation for reference ](https://docs.labelbox.com/reference/import-text-annotations)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "ontology_builder = lb.OntologyBuilder(\n",
        "    tools=[\n",
        "        lb.Tool(tool=lb.Tool.Type.BBOX, name=\"bbox_video\"),\n",
        "        lb.Tool(tool=lb.Tool.Type.POINT, name=\"point_video\"),\n",
        "        lb.Tool(tool=lb.Tool.Type.LINE, name=\"line_video_frame\"),\n",
        "        lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION, name=\"video_mask\"),\n",
        "        lb.Tool(\n",
        "          tool=lb.Tool.Type.BBOX, name=\"bbox_class\",\n",
        "          classifications=[\n",
        "            lb.Classification(\n",
        "              class_type=lb.Classification.Type.RADIO,\n",
        "              name=\"checklist_class\",\n",
        "              scope = lb.Classification.Scope.INDEX,\n",
        "              options=[\n",
        "                lb.Option(value=\"first_checklist_answer\"),\n",
        "                lb.Option(value=\"second_checklist_answer\")\n",
        "              ]\n",
        "            )\n",
        "          ]\n",
        "        ),\n",
        "        lb.Tool(tool=lb.Tool.Type.RASTER_SEGMENTATION,\n",
        "                name=\"mask_with_text_subclass\",\n",
        "                classifications=[\n",
        "                    lb.Classification(\n",
        "                        class_type=lb.Classification.Type.TEXT,\n",
        "                        name=\"sub_free_text\")\n",
        "                    ]\n",
        "                )\n",
        "    ],\n",
        "    classifications=[\n",
        "        lb.Classification(\n",
        "            class_type=lb.Classification.Type.CHECKLIST,\n",
        "            name=\"checklist_class\",\n",
        "            scope = lb.Classification.Scope.INDEX, ## Need to defined scope for frame classifications\n",
        "            options=[\n",
        "                lb.Option(value=\"first_checklist_answer\"),\n",
        "                lb.Option(value=\"second_checklist_answer\")\n",
        "            ]\n",
        "        ),\n",
        "        lb.Classification(\n",
        "            class_type=lb.Classification.Type.RADIO,\n",
        "            name=\"radio_class\",\n",
        "            scope = lb.Classification.Scope.INDEX,\n",
        "            options=[\n",
        "                lb.Option(value=\"first_radio_answer\"),\n",
        "                lb.Option(value=\"second_radio_answer\")\n",
        "            ]\n",
        "        ),\n",
        "         lb.Classification(\n",
        "              class_type=lb.Classification.Type.RADIO,\n",
        "              name=\"nested_radio_question\",\n",
        "              options=[\n",
        "                  lb.Option(\"first_radio_answer\",\n",
        "                        options=[\n",
        "                            lb.Classification(\n",
        "                                class_type=lb.Classification.Type.RADIO,\n",
        "                                name=\"sub_radio_question\",\n",
        "                                options=[lb.Option(\"first_sub_radio_answer\")]\n",
        "                            )\n",
        "                        ]\n",
        "                  )\n",
        "              ]\n",
        "        ),\n",
        "        lb.Classification(\n",
        "          class_type=lb.Classification.Type.CHECKLIST,\n",
        "          name=\"nested_checklist_question\",\n",
        "          options=[\n",
        "              lb.Option(\"first_checklist_answer\",\n",
        "                options=[\n",
        "                  lb.Classification(\n",
        "                      class_type=lb.Classification.Type.CHECKLIST,\n",
        "                      name=\"sub_checklist_question\",\n",
        "                      options=[lb.Option(\"first_sub_checklist_answer\")]\n",
        "                  )\n",
        "              ]\n",
        "            )\n",
        "          ]\n",
        "        ),\n",
        "        lb.Classification(\n",
        "          class_type=lb.Classification.Type.RADIO,\n",
        "          name=\"radio_class_global\",\n",
        "          options=[\n",
        "                lb.Option(value=\"first_radio_answer\"),\n",
        "                lb.Option(value=\"second_radio_answer\")\n",
        "            ]\n",
        "        ),\n",
        "        lb.Classification(\n",
        "          class_type=lb.Classification.Type.CHECKLIST,\n",
        "          name=\"checklist_class_global\",\n",
        "          options=[\n",
        "                lb.Option(value=\"first_checklist_answer\"),\n",
        "                lb.Option(value=\"second_checklist_answer\")\n",
        "          ]\n",
        "        ),\n",
        "        lb.Classification(\n",
        "          class_type=lb.Classification.Type.TEXT,\n",
        "          name=\"free_text\"\n",
        "        )\n",
        "    ]\n",
        ")\n",
        "\n",
        "ontology = client.create_ontology(\"Video Annotation Import Demo Ontology\",\n",
        "                                  ontology_builder.asdict(),\n",
        "                                  media_type=lb.MediaType.Video)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Step 3: Create a labeling project \n",
        "Connect the ontology to the labeling project."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "project = client.create_project(name=\"Video Annotation Import Demo\",\n",
        "                                    media_type=lb.MediaType.Video)\n",
        "\n",
        "## connect ontology to your project\n",
        "project.setup_editor(ontology)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Step 4: Send a batch of data rows to the project"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "batch = project.create_batch(\n",
        "  \"first-batch-video-demo2\", # Each batch in a project must have a unique name\n",
        "  global_keys=[global_key], # A paginated collection of data row objects, a list of data rows or global keys\n",
        "  priority=5 # priority between 1(Highest) - 5(lowest)\n",
        ")\n",
        "\n",
        "print(\"Batch: \", batch)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Step 5: Create the annotations payload \n",
        "Create the annotations payload using the snippets of code above.\n",
        "\n",
        "Labelbox supports two formats for the annotations payload: NDJSON and Python Annotation types."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "#### Python Annotation Types"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "label = []\n",
        "annotations_list = [\n",
        "          checklist_annotation,\n",
        "          radio_annotation,\n",
        "          bbox_annotation,\n",
        "          frame_bbox_with_checklist_subclass_annotation,\n",
        "          bbox_annotation_1,\n",
        "          bbox_annotation_2,\n",
        "          point_annotation,\n",
        "          polyline_annotation,\n",
        "          global_checklist_annotation,\n",
        "          global_radio_annotation,\n",
        "          nested_checklist_annotation,\n",
        "          nested_radio_annotation,\n",
        "          text_annotation,\n",
        "          cp_masks\n",
        "      ]\n",
        "\n",
        "for annotation in annotations_list:\n",
        "    label.append(\n",
        "        lb_types.Label(\n",
        "            data=lb_types.VideoData(global_key=global_key),\n",
        "            annotations = annotation\n",
        "        )\n",
        "    )\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "#### NDJSON annotations\n",
        "Here we create the complete `label_ndjson` payload of annotations. There is one annotation for each *reference to an annotation* that we created above."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "First, let\"s update the bbox with nested classifications with the corresponding featureSchemaId"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "label_ndjson = []\n",
        "\n",
        "annotations_list_ndjson = [\n",
        "    point_annotation_ndjson,\n",
        "    bbox_annotation_ndjson,\n",
        "    polyline_frame_annotation_ndjson,\n",
        "    frame_checklist_classification_ndjson,\n",
        "    frame_radio_classification_ndjson,\n",
        "    nested_radio_annotation_ndjson,\n",
        "    nested_checklist_annotation_ndjson,\n",
        "    frame_bbox_with_checklist_subclass_annotation_ndjson,\n",
        "    global_radio_classification_ndjson,\n",
        "    global_checklist_classification_ndjson,\n",
        "    text_annotation_ndjson,\n",
        "    bbox_frame_annotation_ndjson,\n",
        "    bbox_frame_annotation_ndjson2,\n",
        "    video_mask_ndjson_bytes_2\n",
        "]\n",
        "\n",
        "for annotation in annotations_list_ndjson:\n",
        "  annotation.update({\n",
        "      \"dataRow\": {\n",
        "          \"globalKey\": global_key\n",
        "      }\n",
        "  })\n",
        "  label_ndjson.append(annotation)\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Step 6: Upload annotations to a project as pre-labels or completed labels\n",
        "For the purpose of this tutorial only run one of the label imports at once, otherwise the previous import might get overwritten."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "#### Model-Assisted Labeling (MAL)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Upload MAL label for this data row in project\n",
        "upload_job_mal = lb.MALPredictionImport.create_from_objects(\n",
        "    client = client,\n",
        "    project_id = project.uid,\n",
        "    name=\"mal_import_job-\" + str(uuid.uuid4()),\n",
        "    predictions=label)\n",
        "\n",
        "upload_job_mal.wait_until_done()\n",
        "print(\"Errors:\", upload_job_mal.errors)\n",
        "print(\"Status of uploads: \", upload_job_mal.statuses)\n",
        "print(\"   \")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "#### Label Import"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# For this demo either run MAL or Ground truth import, not both.\n",
        "\n",
        "# upload_job_label_import = lb.LabelImport.create_from_objects(\n",
        "#     client = client,\n",
        "#     project_id = project.uid,\n",
        "#     name = \"label_import_job-\" + str(uuid.uuid4()),\n",
        "#     labels=label\n",
        "# )\n",
        "\n",
        "# upload_job_label_import.wait_until_done()\n",
        "# print(\"Errors:\", upload_job_label_import.errors)\n",
        "# print(\"Status of uploads: \", upload_job_label_import.statuses)\n",
        "# print(\"   \")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Optional deletions for cleanup"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Delete Project\n",
        "# project.delete()\n",
        "#dataset.delete()\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    }
  ]
}